A Global Search Architecture

Recent advances in communication and storage technology make available vast quantities of on line information But this information is of limited use unless it can be searched e ectively Huge scale and heterogeneity of data raise a unique combination of architectural issues that must be addressed to support e ective search These issues occasion the use of multi user distributed search databases with the following capabilities e cient structured searching of the contents of les having various schema continuous availability in spite of failures and maintenance high throughput incorporation of a continuous stream of updates especially the arrival new data and removal of obsolete data We present an architecture that embodies solutions to speci c technical problems that arise in the provision of these capabilities The global data abstraction Until recently the availability of useful information was constrained by the capacity of physical storage and communicationmedia While this capacity has ceased to be a constraint the availability of useful information is limited by our ability to nd it A number of architectural principles arise in addressing the problem of nding useful information when we assume that everyone has the capacity to access all the world s data at all times Multiple users It is not reasonable to assume that each user has a copy of all useful information No matter how vast the capacity of a single user to collect and store information it will be exceeded by the world s capacity It may be argued that the user could elect to store only information useful to him or her but e ecting this selection begs the question raised here how to nd useful information Multiple servers It is not reasonable to assume that any server has the capacity to handle all the data To a ord scalability the data must reside on several servers that collaborate to solve search problems posed by users Heterogeneous data It is not reasonable to expect that the world s data be converted to a common format or organization for the purpose of searching Data exist in a variety of formats and a variety of repositories It must be possible to integrate these data and repositories into a common architecture while harnessing their unique characteristics for search and retrieval Navigational and non navigational structured search Navigational access such as browsing is a useful technique for exploring data according to a pre de ned organizational structure Creating maintaining and navigating this structure imposes a burden on the user and archivist alike It is not reasonable to assume that it is feasible to create a navigational structure amenable to all users search requirements Therefore it must be possible to nd data without resorting to navigation Nevertheless it must also be possible to harness whatever structure is available to guide the search Continuous availability The universe constantly acquires new data It is unlikely that growth in on line storage and communication capacity will accommodate the addition of all new data unless a comparable quantity of data is deleted It is necessary that the architecture accommodate this ux without massive reorganization It is reasonable to assume that platform components undergo a similar ux at any given time some part of the system will be subject to failure or maintenance new components will continuously be added and old components removed The abstraction of a continuously available universe of data must be preserved it is not reasonable to require the user to circumnavigate failures or to accommodate to system reorganization The abstraction of a global searchable data repository will never be fully realized However it suggests design decisions that yield an architecture with wide applicability The obvious application is to improve our ability to search data on the World Wide Web The current Web architecture is heavily navigational exhibiting a number of the problems cited above On the other hand specialized search databases even those reachable via the Web do not well support the browsing and heterogeneity of the Web Other applications include more specialized large applications like bulletin boards newspapers and archival publications In support of the abstraction we have designed an architecture and built the key components of a multi user multi server search database system This paper reports on the overall architecture and three critical components a new query language supporting non navigational search of data with heterogeneous organization a search engine embodying new e cient algorithms for searching text and for on line update a coded redundancy strategy that maintains availability in the face of failures and maintenance The global search architecture